Mobile IO Input and Output for Smartphones, Tablet, and Wireless Devices including Touch Screen, Voice, Pen, and Gestures

ABSTRACT

A cellular telephone or mobile device with several methods of touch, voice, and gesture based input is described. A user is able to interact with a touch screen display on the device to select one or more keys, expand keys, and customize legends or toolbars. A user may deliver spoken audio to the device and actuate a series of commands on the device including search on the device, search on the Internet, accessing an Internet resource, or downloading a document. A user may be able to execute one or more macros on the device. The device may further have a plurality of authentication methods based on the activities on the device. Authentication requests may be made for specific applications or the device. An authentication request may include identifying pictures taken on the device and engaging in activities with the photos.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 13/804,942, filed on Mar. 14, 2013, entitled Mobile IO Inputand Output for Smartphones, Tablet, and Wireless Devices Including TouchScreen, Voice, Pen, and Gestures. The disclosure of U.S. patentapplication Ser. No. 13/804,942 is herein incorporated by reference inits entirety.

BACKGROUND OF THE INVENTION

The input and output (I/O) methods for electronic devices are limitedand may be difficult to use on tablet computers, smartphones, cellulartelephones and various mobile devices. Effective input and outputmethods are relevant for authentication, command, and control functionsof the device. Keyboards, keypads, and other touch based input andoutput methods on a cellular telephone or mobile device (CT/MD) may bedifficult to actuate due to the size of a key and the size of a person'sfinger or pen-based stylus. Entry on such a device may result in theincorrect key selected due to a misalignment between the user's fingeror pen and the screen. The present disclosure relates to mobile devicesincluding cellular telephones, smart phones, tablet computers, hybridtablet and smart phones, wireless devices, or handheld computingdevices. The devices may include a plurality of radios for wirelesslocal area network and wide area network communication including WiFi,802.11 (any variant), Bluetooth, NFC, or other communication means.

Further, individuals using CT/MDs may find themselves in situationswhere it is difficult to devote their full attention to interacting withthe CT/MD. The lack of attention to the operations on the CT/MD mayfurther result in difficulty with input and output and incorrectoperations. A lack of attention may lead to inadvertent dialing on acell phone. Alternatively, when too much attention is dedicated to atask such as authentication it may become cumbersome, monotonous, andrepetitive. Therefore, a diversity of authentication methods is neededfor greater security and engagement.

SUMMARY OF THE INVENTION

It is an aspect of the present disclosure to enable more efficient andrich input and output on smartphones, cellular telephones, tabletcomputers, and various mobile devices. The CT/MD may include thefunctions of a PDA. The device may use a plurality of functions on thedevice including an image camera, front facing camera, back facingcamera, video camera, eye tracking software, microphone, and fingerprintreader. The device may be capable of multimedia play including video,music, games, voice recording, video recording, still picture capture,streaming music, streaming video, spreadsheet applications, note taking,banking applications, and other productivity tools.

It is yet another aspect of the disclosure for the device to beconfigured with a touch sensitive display. The touch sensitive displaymay be comprised of layers of sensors and protective glass, protectiveplastic, LCD, Super LCD, OLED, AMOLED, Super AMOLED, Backlit LED, colorfilters, polarizers, active matrix systems, in plane switching, or TFT.The touch screen display may be configured with a setting that adjuststhe sensitivity of the display and turns on multi-touch.

It is an aspect of the present disclosure to enable real time photo andvideo acquisition and real time sharing of the photos and videos with aserver for processing, storage, and sharing to other CT/MDs, servers orcloud. The pictures and videos may be edited and processed on the CT/MDor server. The CT/MD or server may recognize the photos, crop theimages, filter the images, enhance lighting, or perform other actions.

Predictive Text System

It is an aspect of the present disclosure to enable an input and output(I/O) system beyond simple predictive text entry systems to includeperson, context, application in use, dialect, prior words entered, orhistorical word patterns associated with a user to auto-correct,suggest, or select certain letters. The system may also analyze spokenvoice from a user based on similar factors to appropriately convertaudio to text and suggest words or sentences. The device may download athesaurus or dictionary or query a dictionary or thesaurus located on aserver in delayed or real time as a user is typing on the CT/MD toprovide suggestions to the user. The device may query a server in realtime to display thesaurus related output as a user types.

The server may collect and use the responses and corrections from aplurality of users and devices to improve the predictive text system.The predictive text system may connect to a crowd-sourced predictivetext database that may be localized for certain regions or countries.The database may map incorrect spellings to suggested words to acceptedor used words. The database may score the number of times that anincorrect word is mapped to the suggested and accepted words to refinethe database. As an example, the server may store in memory therecommendation of a suggested word to a user on a CT/MD. If the user inturn chooses not to change the word, the server may mark this entry asnot having been suggested and not changed. If a user chooses to changethe response, the server records the suggested word or words and theultimate word entered. The server may record the time, location, entiresentence, paragraph, or words immediately preceding and after theoperative word in the predictive text system. The server may record thespecific name and contact information in a document, email, or textmessage to determine a mapping between a suggested word and the contact.As an example, an individual may use colloquial language with closecontacts but more formal language with coworkers. The predictive textsystem may create a specific data file for contacts classified invarious categories and suggest the appropriate words based on thesecategories.

It is an aspect of the present disclosure to allow an individual to usean input and output method that includes predefined sentences andphrases to allow for easy composition of messages. The device may alsodisplay by default a keyboard specific to the contact. When composing amessage to a contact who uses English to communicate may bring up bydefault an English keyboard. A contact who uses Japanese to communicatemay bring up a Japanese keyboard by default. A single CT/MD maydynamically switch between displaying these keyboards based on thecurrent contact being viewed. These keyboards may further includepre-defined, frequently used phrases, or statements customizedspecifically for the contact, time of day, and location. For example ifa given contact is a parent, the keyboard or area above the keyboard maydisplay phrases such as “I will be home in 30 minutes.” This area maysuggest a variety of words, phrases, partial sentences, full sentencesor paragraphs. These suggestions may be learned automatically by forexample a machine learning system based on phrases from the specificuser in their discussions with the user they are presently dialogingwith or from all users they dialog. Language preferences for eachindividual may be based on the country code or area code in a phonenumber, stated preferences of the individual, or a database stored onthe CT/MD or server.

Gestures and Touch for UI Control and Authentication

It is an aspect of the present disclosure to enable the touch basedsystem to be fully configurable by a user, administrator, or serverdevice. For example, a left swipe motion of a hand on the screen may beconfigured to shift a window off the screen to the left. A right handmotion may be configured to snap a window to a predefined portion of theright screen. Other shapes can be configured by the user using a macrosystem. The CT/MD may present a shape drawing window. In the shapedrawing window, a user may perform a gesture either by making contactwith the display of the device or by performing a motion in the air asit is recorded by the camera on the CT/MD or other device. This gesturemay then be saved in the memory of the device or on a server. The usermay link the gesture to a set of macro commands or instructions.

For example, if a user drew a circle as a gesture, then user could linka circle to the movement of a current window to the background and abackground window to the center foreground. Each window could beassigned a position in this roundtable like format. Repeatedly drawing acircle can move through the set of windows in a clockwise or counterclockwise fashion based on the direction of the gesture recorded.Alternatively performing a simple rotation with a hand may in turnadvance the next window in the roundtable format.

A user may create a triangle shape. As an example, a triangle shapecould then be linked to displaying three windows, where a first windowoccupies the top half of the screen, a second window occupies a bottomcorner of the screen, and a third window occupies the bottom rightportion of the screen.

A user may want four or six windows to appear at the same time on CT/MDscreen. In this case, the user may draw a square with four boxes or sixboxes. A user may also use two hands to align the windows. For example,three fingers pushed in the left direction may set up three windows onthe screen. This action followed by three fingers being moved across thescreen in the opposite direction may open three additional windows belowthe initial three windows. This may create a display with 6 windows. Auser may further want to close a window. A window may be closed bydrawing an X on the touch screen display. Other combinations ofvibration of the device, single taps, double taps, or holding of tapsmay be used. These gestures may be further linked to macros which accessservers to carry through a series of functions. For example, a diamondshape may be recorded by a user and linked to saving the currentdocument, checking for new email, querying a news information item orinitiating a phone call to a predetermined contact or emergency number.

Authentication, Prevention of Inadvertent Dialing and Privacy Controls:

It is another aspect of the disclosure to require authentication to thedevice using an interaction experience between a user and the touchsensitive display. The authentication may require a user to engage in anumber of handwriting activities. The user may handwrite a name, object,symbol or reference on the display. The user may actuate contact withthe display through a touch of a finger, pen, or other input method. Theuser may trace an object outline to identify a pattern. A handwritingsample from the user may be stored on the device or on an externaldevice or a server. A handwriting authentication request may be storedon a server and downloaded by the CT/MD on a periodic basis. Theauthentication request may consist of a variety of puzzles, handwritingtasks or answers that a user may need to complete to gain access to theCT/MD.

The authentication request may require an individual to use handwritingor drawing to fill in shapes or pictures or trace images. The drawingmay be done by using a finger against the touch screen or a pen inputdevice. The pen input device may be housed within the CT/MD.

It is another aspect of the disclosure for a user or administrator todevelop a series of authentication activities. A user may createmultiple authentication responses, store these responses on the deviceor a server. These authentication responses may be stored in a look uptable. The device may be configured to present a multiple steps to theauthentication.

It is another aspect of the disclosure for the user to respond to anauthentication request that uses pictures, videos, or other content onthe device to access the device. A user may be requested to identify thepeople in an image stored on the device. A user may be requested to beidentify a location including geography that an image taken by a cameraon the device. The image may include location data identify the latitudeand longitude, city, location, date or time of the image capture.

It is yet another aspect of the system provide for multimodal ormultidevice authentication in which a user must execute a series ofauthentication steps across applications, devices, or locations. Theauthentication request can be required to access the CT/MD,configuration settings of the CT/MD, applications on the device, or thesettings of a remote system. As an example, a user may need toauthenticate on a plurality of devices in order to access an applicationon the first CT/MD.

It is yet another aspect of the disclosure to use data from oneapplication on the device in conjunction with data from anotherapplication on the device or on another service or device. Anauthentication service could use the transaction or history data from amusic application, browsing application, email applications, phone callapplication, photo application, video camera device, location, or GPShistory.

It is yet another aspect of the disclosure to allow for a series oficons to be displayed on a screen of the CT/MD prior to the user gainingfull access the CT/MD such that a user may only authenticate to anapplication and note the device. A user may then select an icon toactuate an application or macro. The actuation of the icon cansubsequently request a user to authenticate to the device. A user maythen be authenticated to a sole application or macro function on thedevice instead of authentication to all the applications on the device.A server may manage which applications a user can access, whichapplications require a specific authentication method, and whichapplications a user can install, execute, or delete. The server mayfurther configure, install, delete or execute applications or data onthe device. The server may associate the device, including the device'sunique id, with information specific to that user, a set of permissionsassociated with that user, a list of applications associated with theuser or device in conjunction with a policy and provisioning system.

It is an aspect of the disclosure to allow for a user to authenticate tothe device by selecting a list of contacts that were previously called.The device would present a list to a user of potential contacts calledand the user may be required to select which contact was actuallycalled.

It is an aspect of the present disclosure to prevent users frominadvertently making phone calls prior to a call. A user may be requiredto confirm their phone call after placing the call. This may require auser to see a second message dialog box to confirm the call.

It is yet another aspect of the disclosure to allow for a system ofpreventing inadvertent phone calls, email messaging, or otherapplication use during an active call.

It is an aspect of the present disclosure to enable a screen to belocked upon a call being initiated if a specific configuration settingis enabled. The phone may lock the screen to disable touch input to thedisplay in response to movement of the phone. The movement of the phonemay be acquired using an accelerometer located in the device. The devicemay use user interaction to determine when to lock the screen ortemporarily disable touch based contact. A user that initiates a callmay then move the phone to the ear. The device may first determine thata call has been moved. The device may then detect movement for aduration or period of time post call initiation. Upon the receivingparty accepting the call, the device may then lock the screen ortemporarily disable touch based input. The device may also secure thescreen from other activities or disable touch based input based on anambient light sensor. When the available light detected by the ambientlight sensor is low, for example, when the device is held next to thecheek of a person's face, the device may disable the touch screen orautomatically secure the screen. The device may also be configured by aserver or by the individual with the time or events that lead to thescreen being locked. A user may physically shake the device to callsomeone, physically shake the device to terminate a call, or physicallyshake the device to lock the screen.

It is yet another aspect of the disclosure to prompt a user of apotential inadvertent phone call by detecting the behavior of the user.In one example, if a phone call has been in session for an extendedperiod of time and there has been no spoken words received by themicrophone on the device, the device may detect a likely event of aninadvertent phone call. The device may then signal a warning that aninadvertent phone call has occurred by making a sound, a beep, orterminating the call. The device may use noise cancellation and noiseidentification to determine whether spoken audio or general backgroundnoise was determined. If only background noise is detected and no spokenaudio, the system may conclude that the call is an inadvertent call,notify the user, and terminate the call.

By linking the drawing of a shape on the screen to a predefined action,the user may circumvent prying eyes from awareness of the activities ofthe primary user on the device. A witness to the CT/MD may notunderstand what actions a given gesture may be linked to. As an example,an individual that is in an emergency may not have the ability to call‘911’ or another public authority. The user could draw a star shape onthe screen and a telephone call, text message, or email message to apublic number, emergency contact, or family member may be initiatedautomatically with a predefined method. This call may further have thespeaker turned off while the microphone continues to record.

It is an aspect of the present disclosure to enable the gestures to beconfigured to only execute when a specific finger is used. A specificfinger or fingers may be configured for the gestures. Many individualsmay use their index finger for touch based activities. In order toprevent incorrect usage of the finger, a different finger such as a ringfinger or pinky finger could be used. The CT/MD or server could identifythe specific finger by the size of actuated area on the touch screendisplay, by a fingerprint reader, by a camera which may record the userperforming the action.

It is an aspect of the present disclosure to enable the CT/MD to bepreconfigured with various shapes as described above or the CT/MD maydownload gestures from a server, or application store/exchange, whichmay accept suggested gestures from a community of participants.

It is an aspect of the present disclosure to enable the CT/MD to beconfigured for an obfuscated displayed. Individuals using mobile devicesmay need a private display which prevents other individuals from viewingthe current screen of the device. This scenario may occur in a varietyof places including trains, planes, and automobiles where an individualmay sit in close proximity to another individual. In this scenario, theCT/MD may display multiple windows simultaneously where a first windowmay display a first set of text and a second window may display a secondset of text. A user may compose a message, which displays in a thirdwindow of the device. The text may be associated to add to the text ineither the first or second window. However, in this privacy mode, thescreen may not redraw the text onto the screen. This can prevent otherindividuals from reading the full document while it is being typed.Instead individuals may only see the text being currently written uponin the third window but not the entire document at the same time.

It is an aspect of the present disclosure to have one or more full orpartial virtual keyboards, keypads, or input mechanism displayed on atouch sensitive display on the device. These keyboards may be splitacross the screen of device and appear in multiple locationssimultaneously.

Notifications:

It is an aspect of the present disclosure to enable the device to beconnected to a server or Internet device to download rules, protocols,keys, icons, menus, toolbars, legends, and data about the icons andmenus. Menus and other UI items for a specific device may stored as aconfiguration setting on a server. A user may configure the locations,size groupings, width between the icons displayed, icon picture, andmenu colors. The server may record these configuration settings toenable multiple CT/MDs to automatically have the same settings. Thedevice may communicate with the server to download and updateconfiguration settings. The server may control the configurationsettings, locations of icons, menus, and other user interface items onthe device. The server may mirror the configuration settings and userinterface items across multiple devices. The device may be identified bya unique device ID which may be associated with configuration, security,and application management policies.

It is an aspect of the CT/MD to provide a notification service to thedevice. The CT/MD may download updates including information messages,meta data, application data, rules, and other items from a server. TheCT/MD may be configured to frequently poll a server for the information.The Server may automatically send this information to the CT/MD. Thenotification information may include new emails that may have arrived,text messages, application relevant notifications, or operating system(OS) notifications such as OS updates. The icons may be modified by aserver including appending a number next to the icon, changing the imageof an icon, changing the color of an icon, or making the icon blink.Similar modifications may be enabled for other UI components based onnotification information.

It is yet another aspect to enable to the CT/MD to download from theserver a new menu, icon, or update all notifications upon entering aspecific location as determined by GPS, WiFi access point, or otherlocation method. Alternatively, a server may send information to thedevice to update the display of an icon including a numeric indicatornext to an icon.

Movable UI, Keyboards and Windows on Mobile Devices:

The CT/MD may download a keyboard or keypad from a server. The devicemay be configured with a set of popular keyboards by default. Thekeyboards may be enabled in one or more languages. The keyboard may beenabled with one or more symbols or emoticons. The keyboard may beconfigured for input to a specific appliance. As an example, for a CT/MDto interact with a television may require a separate keyboard whichincludes up/down channel indicators, volume indicators, numeric entryfor a channel, or a PIP capability. An intelligent appliance such as amicrowave may require a keyboard which accepts input of time of cooking,power setting or type of food.

It is an aspect of the present disclosure to allow for the device toplay a sound in response to a key being depressed. Alternatively, thedevice may vibrate in response to a key being depressed. A user maydesire a key to be read aloud as it is depressed. A key may also behighlighted in an increased font size or window size as it is depressed.

It is yet another aspect of the present disclosure to have an on-screenor virtual key may further be mapped to communicate automatically with aseparate device. As an example, a key on the right side of the CT/MD maycorrespond to changing the channel on a intelligent television. A key onthe left side of the device may correspond to turning on or off a DVRsystem.

It is an aspect of the present disclosure to have a physical key mappedto more than one function on the device. A physical key on the devicemay be reprogrammed by an individual on the device or by a setting on aserver. A physical key on the device may be configured by an applicationstored in the memory of the device. A physical key may correspondinglybe enabled with a plurality of functions based on the application orcontext. The context may be include the lighting in the room,temperature, the applications running on the mobile device, the locationof the CT/MD as determined by Global Positioning Service (GPS), Wi-Fi,802.11a,b,c, n (or any variant), Bluetooth, or other protocol, orproximity to other mobile devices. A server or a user on the device maycontrol the assignment of a physical key to a specific action, set ofactions, or applications. A physical key may further be mapped tocommunicate automatically with a separate device. As an example, a keyon the right side of the CT/MD may correspond to changing the channel ona intelligent television. A key in another location may correspond tochanging the volume on an intelligent television. A key on the left sideof the device may correspond to turning on or off an Internet enabledmusic device.

In another embodiment of the present disclosure, the keyboard may bedisplayed in a fixed or movable area on the device. The keyboard may beconfined to a specific window. A user may drag and drop a keyboardaround the device to place it a desired location. Alternatively, a usermay drag specific keys or macros to be placed in certain locations onthe screen. An ‘enter’ key which be frequently used may be placed on theright side of the device. Alternatively, the ‘enter’ key could be mappedto a specific gesture such as a down arrow on the a specified side ofthe screen. Alternatively, text may further be moved around the screenof the device. These items may then be placed to always appear in aparticular region such as the top right, bottom right, or other area ofthe screen.

It is an aspect of the present disclosure to display the keyboard in oneor multiple areas of the CT/MD simultaneously. The keyboard may beconfigured to be a fixed location on the CT/MD. It is further an optionfor the keyboard to be configured in a variable position on the CT/MD.It is an aspect of the present disclosure to enable the keyboard to bein a floating position on the CT/MD. A user may drag the keyboard from abottom area of the device to the top of the device such that keys arelocated on the top of the device and the other viewable content is shownbelow the top of the keyboard.

It is an aspect of the disclosure to enable a full or partial keyboardto be associated with the right or left side of the device. A numerickeypad may appear on the right side of the display and the content mayappear to the left side of the display. Alternatively, the numeric keysmay be split in various sides and corners of the device while the textkeypad is situated in the center of the screen.

It is an aspect of the present disclosure to allow for physicalvibration of the device in response to a key, icon, menu, legend orother item being selected on the CT/MD. A vibration of the device mayprovide an output mechanism important to individuals that aredistracted. A sound may be played or a letter may be spoken after a keyis depressed. A letter e′ maybe spoken aloud by the device. A fullsentence may also be spoken by the device. This may aid in input andoutput for visually impaired.

VOICE TO TEXT, DICTATION, AND SERVER UPDATES

It is an aspect of the present disclosure to allow the user to dictate amessage to the CT/MD and display the content of the dictation on thescreen on the CT/MD. The local device or a server may perform the speechto text conversion. The server may stream the results in near real timeas the conversion is being taken place or send a set of converted dataafter a certain time period or amount of content. As an example, aserver may send an entire converted paragraph of spoken audio into textin order to reduce the overhead of the packet message being sent. Theserver could send the data in real time which may increase the overheadof the messaging but improve the speed with which the results aredelivered. These preferences may be configured on the CT/MD or on theserver.

It is an aspect of the present disclosure to allow processing of voiceto text to occur on the CT/MD or a server. The server may deliver thetext in an encrypted format back to the device. The server may improveits voice to text translation based on the spoken patterns specific to aperson, classification of a person based on demographic information suchas age, or based on other users of the system.

It is an aspect of the disclosure to have the dictated message stored ona remote server.

It is an aspect of the present disclosure to provide information boxes,windows, or dialog messages to enable enhanced operating system to userinteraction, software application to user interaction, remote device touser interaction, and server to user interaction. This interaction mayenable a user to be prompted with a message box on the CT/MD to confirmactions based on data from a remote server, Internet server, or networkdevice. These message boxes may be controlled by the server, configuredin response to server messages, or defined by a software program on thedevice.

It is an aspect of the present disclosure to allow for an icon to beconfigured as a macro or be configured to execute a series of functions.The icon may be structured as a shortcut to perform several functions insequence, simultaneously, or delayed. An icon can be configured to calla person, launch an application, launch a function within anapplication, execute a series of commands in one application, execute aseries of commands in multiple applications, query an Internet resource,complete a series of Internet actions including performing a search andclassifying a response.

The macro keys may be defined by the user by selecting a series ofoperations to allow. For example, a user may create a macro to send amessage to a contact using all available communication methods includingemail, SMS, and voicemail. The macro keys may allow for data to beshared from one application to the next application. A photo in oneapplication may be automatically sent to a photo imaging program forprocessing and refinement and then returned to the photo application.The server may also prevent certain macros or sequences of commands fromtaking place on the device. The device, for example, may be prevented ofsharing data from one application to another application or to aspecific server. The macros may be executed in response to a touch orvoice or audio input.

It is yet another aspect of the disclosure to enable multiple inputsincluding touch by a finger, pen or stylus or audible input includingspoken commands, voice, sound simultaneously. As an example, the CT/MDmay be configured for full multitasking across applications and I/Osystems.

It is an aspect of the present disclosure to allow for the actuation ofone icon to create a connection with a remote server. As an example, asingle icon or button can execute a series of actions to update thecontent, applications, media, preferences and settings on the CT/MD witha server or servers. This icon may provide a simple method forsynchronization across several applications.

In yet another aspect of the disclosure, a user may be prompted with adialog box or message window to confirm whether the user would like tocarry through with the call. This second confirmation may reduceinadvertent calling. This requirement may be for specific contacts inthe CT/MD address book.

It is yet another aspect of the disclosure the device may be configuredwith compressed or expanded keys, keyboards, menus, buttons, icons, orother graphical user interface items. A given key may be expanded whenit is initially pressed or while it is being depressed. Alternativelythe items may be expanded or compressed based on the orientationincluding portrait, landscape, or inverted of the device. The size mayalso be configured by the individual as a preference, a devicepreference or by a server preference.

It is another aspect of the disclosure for the device to changefunctions based on available battery life on the device. As an example,the device may turn off or reduce the quality of all video applicationswhen the battery reached below a certain threshold. The device mayreduce the fidelity of the images displayed including the brightness andupdates to pixels based on the battery life.

It is yet another aspect of the device to enable the keyboards, menus,legends, and icons to be updated dynamically by a server or in responseto data from a server. As an example, the keyboard may be updated when anew device is discovered by the CT/MD.

It is yet another aspect of the disclosure for the device to change theprimary keyboards, keypads, legends or menu items in response to a touchbased input. As an example, a user may execute a series of steps tosplit the screen on the CT/MD and simultaneously make available alegend, keyboard, and text document view.

It is an aspect of the disclosure for the device to show the desktop ofthe device by a user tapping a predefined area of the CT/MD. It is anaspect of the CT/MD to display half a screen of the desktop in responseto a user actuating a part of the display. Alternatively, touching anicon or menu bar may also result in showing the desktop of the CT/MD.

It is another aspect of the present disclosure to display a new keypad,legend or area based on the application that is running in theforeground of the CT/MD. As an example, a software application thatrelates to telephony by default shows a numeric keypad while anapplication focused on message composition by default shows a standardkeyboard.

It is yet another aspect of the disclosure to allow for a toolbar,navigation bar, or legend bar to be fully configurable. The toolbar,navigation bar, or legend bar may be configured to include other icons,shortcuts, or macros. A user may configure which icons to place in thetoolbar, navigation bar, or legend by moving an icon from one displayarea to the toolbar, navigation bar, or legend. A user may add icons toa toolbar located on the bottom of the device. A user may scan acrossitems in the menu while the menu is in a fixed area of the display. Thearea outside of the menu bar may stay constant. The menu may contain aseries of keyboards, icons, phrases, words, or other items.

It is yet another aspect of the disclosure to provide an overlay imageon top of other images on the device for authentication purposes. Theauthentication method may be associated with a sequence of sounds. Akeyboard may be displayed which may be associated with a musicalinstrument keyboard. For example, a keyboard may include the keys of aviolin, piano, or other musical instrument. The sequence of notes playedmay also provide an authentication method to access an application, suchas a music player, or the device. The use of authentication methods tiedto the nature of the application may make the authentication request andprocess more relevant to the user.

Other aspects, features and advantages of the present disclosure willbecome apparent from the following detailed description when taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The accompanying drawings, being incorporated in and forming a part ofthis specification, illustrate embodiments of the disclosure and,together with the description, serve to explain the principles of thepresent disclosure:

FIG. 1 is an embodiment showing an example of an authentication requeston the screen of the device using a history of the interaction of acamera application.

FIG. 2 is an embodiment showing an example of an authentication requeston the screen of the device based on the call history.

FIG. 3 is an embodiment showing an example of a multi-levelauthentication request.

FIG. 4 is an embodiment showing an example of an authentication requestin conjunction with a CT/MD and server.

FIG. 5 is an embodiment showing an example of sending data to the CT/MDand receiving data from the CT/MD.

FIG. 6 is an embodiment illustrating an example of collaboration acrossmultiple devices using text, signature, and graphics.

FIG. 7 is an embodiment showing an example of CT/MD with multiplekeyboards spaced at different positions on a touch screen display.

DETAILED DESCRIPTION OF THE INVENTION

The present disclosure enables a key on the device may be expanded basedon a user actuating a region of the touch screen CT/MD. An applicationon the CT/MD may expand a key on the CT/MD in order to highlight aspecific key that is depressed or should be depressed. A key may beexpanded in size by a configuration setting on the CT/MD. The fullkeyboard may be expanded or compressed when a first key is actuated. Asecond key may be expanded in response to a first key being expanded tofacilitate easier actuation of another key. The second key may be anexpanded version of the first key and display simultaneously with thefirst key being displayed. A second key may be expanded if it is asuggested next character.

The present disclosure further allows users to see a variety ofcompressed or expanded keys based on configuration settings. A user mayenter a desired key expansion amount or set a percent expansion of akeyboard.

The present disclosure further allows individuals to use a variety ofcompressed or expanded keys based on contextual or situational factors.A key may be expanded based on the ambient light detected by a lightsensor on the device. In periods of high intensity light, keys may beexpanded to take up a greater area of the viewable screen to facilitateeasier viewing. A key may also be expanded in response in response tomoisture, which may result from perspiration by the individual orhumidity.

The present disclosure further allows a user's attention to be directedto certain activities and actions. A key, button, or icon on the CT/MDmay flash to attract a user's attention. A key, button, or icon may behighlighted in a first or second color to attract a user's attention andserve as a notification. Notification messages may be flashed on thescreen or appear translucent on the screen. A selected key may appear ina first color and the full keyboard may appear in a second color, appearin a color of less intensity or appear translucent.

It is an aspect of the present disclosure to enable the server toinfluence or control the keyboards, menu items, icons, notifications,and toolbars on the device. The device may be configured to change anicon, keyboard, menu item, or legend in response to a message, data, orother information received from a server.

There may be a hierarchy of icons or macros configured in the device.Icons maybe group together into one or more menus. For example, two orthree icons may be associated together and placed into a group or menu.This group or menu may be given a new name or icon for actuation. Thisset may then be grouped with additional icons. Icons may therefore becascaded in a hierarchy of use on the device. A user may create a newicon using a drawing program. A user may use a photo as an icon. A photomay also be used as a macro function. For example, a first photo of acontact may be placed on the desktop of the device. Actuation of thisfirst photo may call the contact's home number. A second photo of thecontact may be placed on the desktop of the device. Actuation of thissecond photo may call the contact's mobile phone number. These photosmay be similarly associated with a hierarchy. Selecting the first photomay run a macro to execute a series of functions. For example, actuationof the first photo may call the home number and if there is no answermay subsequently call the contact's mobile number, send a text messageor compose an email.

Authentication:

It is an aspect of the present disclosure to enable the device or anetwork server to use multiple authentication methods in conjunctionwith one another to authenticate a user to a device. A user may berequired to satisfy a plurality of input and output methods or pass apercent threshold of authentication methods in order to satisfy anauthentication requirement. A server may track the number ofauthentication methods used in order to grant a match between theinputted data and stored values. The server may require authenticationresponses on multiple devices in order to authenticate a new device orexisting device.

The device may also accept voice based input to a macro. The macrofunction may execute a series of commands. A macro may be spoken to thedevice. The device in response may convert the spoken audio to text or adata command. The command may execute an application on the device. Theapplication on the device may contact a server to process the request,query another server, query an Internet service, search the Internet,download data from a database, or process a calculation. The server mayreturn the raw data to the device or a synthesized response to thedevice. The device may accept a spoken command to access a data file ona server. The device may accept a command to control other devices suchas a garage door opener.

The escrow server may be a identity management server, authenticationserver, digital certificate server, public key/private key encryptionserver, policy server, or other server.

FIG. 1 illustrates the display 701 of a CT/MD 700. The CT/MD 700presents an authentication request on the screen of the device. Thedevice uses the history of the interaction of a camera application todetermine the authentication request 702 to the device. The CT/MD or aserver may record in a table, database or other storage means whichpicture was taken using the camera on the device. Additional meta dataabout the image taken may be accessed from the image itself which may beencoded with location, GPS coordinates, time of day, device used forimage capture, or other information. The CT/MD or server may determinewhich image is received by storing in a data table meta data aboutreceived images. Alternatively, a user may enter in this data in anauthentication setting. Alternatively, a server administrator mayconfigure images for authentication. A user may be asked to identifywhich image was taken 703 using the camera 704 on the device and whichimage was received 705. The user may indicate a response by spoken wordsusing phrases such as ‘image on the left’ or by touching an area of thedisplay 706 corresponding to the correct response using a finger or pen.

FIG. 2 illustrates the display 801 of a CT/MD 800. The CT/MD 800presents an authentication request 803 based on the call history. Thecall history may be stored in memory on the device or a database 804located on the device or on the server. The CT/MD or server may presenta series of questions to the user seeking access to the device aroundwhat calls were made from the device. As an example, a user may be askedwhich of the last 3 numbers or contacts was called or when a call wasinitiated to these numbers. This would correspond to an lower securitylevel since the CT/MD or server is providing information to the user. Amore stringent security setting may require a user to enter in a phonenumber called recently, enter the name of a contact recently called, orselect a contact listed as a favorite or family member. Alternatively, auser may be asked questions around the frequency with which they call aspecific number. This information may be provided in a range (daily,weekly, monthly, rarely) to make it easier for the user to answer thequestion. The call history may also be linked to a location. Forexample, a user may be asked to answer where they often call their homenumber. One of the options may require a user to type in the street nameor city or identify a location on a map. A user may often call home fromthe office but may be unlikely to call home from their home location.These settings may be modified for higher or lower security levels.Questions may further be modified so as to provide information as partof the authentication response or not provide information based on thedegree of security required. Similarly, the authentication requests maybe a secondary authentication request after a higher or lower securityauthentication request. Alternatively, the authentication request mayfor a specific application. A corporate application, email, or creditcard related application may necessitate a higher level of securitywhile a media application may not. A phone number itself may beassociated with an authentication item. As an example, various contactsmay be associated with high or low security levels or variousrestrictions. Individuals may often need a single device instead of awork and personal device. Correspondingly, the work and personalcontacts and numbers may end up on the same device. A user may wish todisable contacts associated work during non business hours. The CT/MDmay allow for a user to enter a separate authentication method forcertain contacts such as office contacts based on the time of day orother setting.

FIG. 3 shows a multi-level authentication request on a CT/MD 900. Anapplication may be executed in response to contact or selection from afinger, pen, or other object made with the icon 901 on the display. Theauthentication request may grant access to both the device 900 and theapplication 904, only the application 904, or only the device 900. Theauthentication request may be based on a history item related to theapplication 904 for which access is requested or for another applicationsuch as messaging 903. A history item may include any activity thatoccurred on the device. As an example, the name, date and time anapplication was installed on the CT/MD may represent a historical item.Alternatively, the last website accessed on the browser may be used as ahistory item.

FIG. 4 illustrates an authentication request in conjunction with a CT/MD1000 and server 1001. The server may request a type of authenticationrequest of the device. An application that requests a streaming musicservice from a server may be required to satisfy an authenticationrequest 1003 defined by the server 1001. The streaming music service maybe initiated by making contact with the icon 1004. The device maypresent the authentication request in a textual or graphical form 1005on the CT/MD 1000. The authentication request 1003 may require the useof historical usage data of device 1000, server 1001 or escrow server1006. The authentication request 1003 may contain information thatrequires a user to enter information on the CT/MD 1000 to confirmphysical possession of the CT/MD 1000. The server 1001 or escrow server1006 may define the number of authentication requests required. Theauthentication request may further require a plurality of authenticationresponses across a second electronic device 1007.

FIG. 5 shows the server 1101 sending data to the CT/MD 1100 andreceiving data from the CT/MD 1100. The icons 1102, 1103, 1104, ormacros 1105 may be configured standalone or in conjunction with theserver 1101. These icons may be modified in real time based on data fromthe device or server. The macros, icons, legends or other items maychange in response to data from the server 1101. After receipt of thedata 1106, the CT/MD 1100 may show different icons 1107 or differentmacros 1108 with pictures, numbers, shapes, highlights, or blinkingindicators. Names of icons 1101 may also change. A physical key 1111 mayalso be redefined based on a configuration setting on the device 1106 oron the server 1101.

FIG. 6 illustrates collaboration across multiple devices using text,signature, and graphics. The server 1200 facilitates the transfer ofdata between multiple CT/MDs 1201, 1202, and 1203. The server 1200receives a message 1204 from a device 1202. The server sends the message1204 to another device 1203. The user may respond to the message on thesame device 1203 or another device 1201 where the response is sent firstto the server 1200 and then to other CT/MDs 1202 and 1203. The server1200 stores the data for the messages in the database 1205. The messagecontent may also include video or images captured using the camera 1206.

Using the camera, the participants may engage in peer to peer videoconferencing or server mediated video conferencing. A fingerprint imagemay be acquired by a fingerprint scanner 1207 located on the device orby a camera located on the device to authenticate a person or newdevice. The CT/MD or server may store digitized samples of finger printdata in a look up table for a person. The CT/MD may also acquire an inkbased image of a fingerprint using the camera and process the image forfingerprint recognition either on the CT/MD or on a server. A new devicemay be synchronized with the server and download the shared data afterproviding a positive response to an authentication query.

The device may be further configured with the ability for a user on afirst device to send a signature request for a document 1209 to acontact in the contact list of the device. The document 1209 may alsoinclude purchase or transaction requests. For example, a purchase ordermay need to be electronically signed. The request may be sent to aspecific device using, e.g., a text message. The document 1209 may alsobe sent to a specific email address.

A user may set a hierarchy of signatures required to complete theapproval of a document. For example, the document may be first requiredto be signed by a first individual. A second or third individual may berequired next. Lastly, a fourth individual may be required to completefinal approval of the document 1209.

A user may define a time period under which a signature 1208 should becompleted.

The document 1209 to be signed electronically and supporting documentsmay be stored on the device or on a server. A request including anInternet accessible link to the document 1209 may be sent from a user ona first device to other users on other devices. A user on a seconddevice may download the document 1209 and select an area of the document1209 to annotate or sign. The document 1209 may be annotated withmarkers to identify areas for the signature. The document 1209 may beannotated with notes. The CT/MD, Server, or other electronic device maycollect the responses, process the signatures including for handwritingrecognition, and notify the other devices when the process is complete.The finalized signed document may not be published to the other devicesuntil all the signatures are collected.

The signature may come from the user actuating with the screen usingstrokes of the finger or a stylus on a specified region for a signature.

A document may require a second authentication method in addition to asignature 1208. The second authentication method may include a pictureof the individual obtained by a camera 1206 on the device, a fingerprintreading 1207, a GPS location, a phone call to an online system, aresponse to a text message with a password or identifier, an email to aspecific address, or another method.

The users may be notified using email, text message, voice message, oranother method when all the signatures are complete.

The CT/MD with the server can enable a plurality of functions includingdocument viewing, sharing, annotating, modifying, editing the textdocument, graphics, image, video, audio, sound and data and orcombinations thereof as a collaborative product 1210. The collaborativeproduct 1210 between users may be inclusive of text, voice, sound,audio, image, video, graphics and data; As an example, a collaborativeproduct may include all the pictures taken on a given day and furtherannotated with comments on each picture.

The collaborative product 1210 may be accessed in a contemporaneousmanner and/or at a later time by one or more users. The collaborativeproduct 1210 may be enabled for maintaining revision control by thecentral server or a designated intelligent keyboard user. Thecollaborative product 1210 may be enabled for time stamping the accessand actions of one or more parties to the collaborative work product inprogress. The server may maintain electronically valid signatures imagesand authentication files for each user that is party to the transactionor signature event.

The CT/MD may feature a signature window and enable witnesses tocorroborate the event. Certain witnesses that have obtained appropriateapprovals may notarize the document electronically. Other witnesses mayhave associated with them various credibility ratings. A witness mayalso sign as to observing the signature in either the physical presenceor by watching the signature in real time in an online fashion. Thewitness may also be recorded using the camera or video camera on theCT/MD. As an example, the video camera on the CT/MD may record or streamthe signing event to another CT/MD or server. The other CT/MD or servermay store the signing event for future playback, auditing, or otherpurposes. Alternatively, the camera on the CT/MD may take snapshotswhile the signature event is taking place. An audio recorder integratedinto the CT/MD may record sounds that are taking place at this time. TheCT/MD may further record the GPS location of the signature event. Thesignature and associated meta-data related to the event may be archivedto the signature document and stored locally on the CT/MD or on theserver.

The collaboration event may further be enabled to record negotiation andrequests by one or more parties. The collaboration product 1210 may bemarked as pending or as a final version. Parties may state by selectinga touch box, an accept button, or other marker to indicate agreement.

The parties privy to the collaboration event may be required to agree tothe event at a specified time, in real time or simultaneously.

The server may compare the signatures, signature images, signaturestrokes, time to sign, signature movement across the touch screen, tostored samples located on the device or a server. The CT/MD or servermay perform handwriting recognition using a software program.

There may exist a plurality of servers enabled to stored one or moresamples of the data. A server may function as an escrow server wherebythe server holds authentication information or collaboration informationfor a specified time or rule.

The authentication system may require a plurality of authenticationmethods simultaneously, sequentially, temporally spaced, orgeographically spaced using GPS information. One authentication methodmay require a voice reading while a second authentication method mayrequire a password being entered on the device. A server may requireauthentication to a service by requiring a user to enter a password onthe CT/MD and a code delivered by the server to the CT/MD. The servermay also require authentication by requiring a user to enter a passwordon the CT/MD and then a password obtained from the CT/MD while thepassword is entered. An image may compared against a stored sample onthe device or on the server. The image may also include a retinal scanthat is compared against a stored sample.

The server or CT/MD may also require and obtain GPS location informationfor the phone during an authentication event. The server may correlate aplurality of authentication information to determine whether thecollaborator is in proximity to another collaborator. The server may usean IP address to determine location. The server may also store andcorrelate IP addresses for authentication purposes.

FIG. 7 illustrates a CT/MD 1300 with multiple keyboards 1301, 1302,1303, 1304, 1305, 1306, 1307, 1308 spaced at different positions on atouch screen display. A user may define the ideal spacing in aconfiguration file. Alternatively the CT/MD or server may contain idealspacing information. An individual's hand size and finger size may beused to create an ideal spacing of the keyboard. The finger size may bemeasured during a calibration process where the user places theirfingers on the touch screen and the CT/MD determines the points ofcontact. Alternatively, a user may describe their hand type such assmall, medium, or large and the system may use the associated keyboardspacing. The CT/MD may also download various keyboard spacing from aserver. This spacing allows for greater vertical separation between therows of a keypad. A horizontal separation may also be enabled ordifferent combinations of split keyboards groups of keys, or specifickeys may be configured. Keyboards 1302 and 1308 feature greaterhorizontal and vertical separation than a full numeric keypad organizedas a row. Similarly macros or icons can be split across different areas.The location of the keyboards may be configurable by a user. Thekeyboards may allow for easy access to keys to prevent inadvertentselection of the wrong character. A first keyboard such as the numerickeyboard 1302 may be located at the top of the display and may appear ordisappear based on the application the CT/MD. The application 1309 asshown is a spreadsheet application where character entry and numericentry may both be required by a user.

The keys in the keyboard typically could be LCDs for displaying therespective legends, and desirably are touch sensitive.

The foregoing descriptions of specific embodiments of the presentdisclosure have been presented for purposes of illustration anddescription. They are not intended to be exhaustive or to limit thedisclosure to the precise forms disclosed, and it should be understoodthat many modifications and variations are possible in light of theabove teaching. The embodiments were chosen and described in order tobest explain the principles of the present disclosure and its practicalapplication, to thereby enable others skilled in the art to best utilizethe present disclosure and various embodiments, with variousmodifications, as are suited to the particular use contemplated. It isintended that the scope of the disclosure be defined by the Claimsappended hereto and their equivalents.

What is claimed is:
 1. A system for gesture based input to amultifunction mobile device with a processor, touch screen display, anda keyboard, the system comprising: a gesture based input, wherein themobile device is enabled to accept gestures and shape input; a splitscreen keyboard, wherein the mobile device is enabled to display a splitscreen keyboard and accept input to one or more applications based ontouch based input on the screen entered at first side and second side ofthe mobile device; and a macro function on the device, wherein saidmacro function comprises executing a plurality of functions.
 2. Thesystem of claim 1, wherein the mobile device is enabled to accept adrawn object shape and enable a phone call to be activated or textmessage to be sent using a predefined method.
 3. The system of claim 1,wherein input to the mobile device is based on the specific finger usedand wherein the mobile device is enabled with a fingerprint reader. 4.The system of claim 1, wherein the mobile device is enabled to customizethe shape of the device area based on the finger size.
 5. The system ofclaim 4, wherein the mobile device is enabled to download gestures froma server or application store.
 6. The system of claim 5, wherein themobile device displays an obfuscated display to accept specified shapebased gestures and to delay redrawing of one or more displays.
 7. Thesystem of claim 1, wherein a mobile device is enabled with a splitscreen keyboard, wherein a portion of the keyboard is placed acrossmultiple locations of the device.
 8. The system of claim 7, wherein thekeyboard is movable to multiple predefined areas on the device screenand wherein the keyboard may be dragged and dropped to various areas onthe screen.
 9. The system of claim 8, wherein a full or partial keyboardis associated with one side of the device and/or wherein the numerickeys may be split in various sides and corners of the device while thetext keypad is situated in the center of the screen.
 10. The system ofclaim 9 wherein the mobile device vibrates based on the key depressed onthe screen and wherein an icon is configured as a macro to execute aseries of functions in sequence, simultaneously, or delayed.
 11. Thesystem of claim 10, wherein the mobile device customizes the displaybased on the context including one or more of the lighting in the room,temperature, the applications running on the mobile device, the locationof the CT/MD as determined by Global Positioning Service (GPS), Wi-Finetwork and/or proximity to other mobile devices.
 12. The system ofclaim 1, wherein the mobile device further enables display multiplewindows simultaneously including a first window for a email applicationand a privacy mode in which a document is partially displayed inresponse to proximity to another individual or device.
 13. The system ofclaim 1, wherein the mobile device is enabled to accept a gesture basedinput and wherein said gesture based input is a shape drawn in the airin proximity to the mobile device.
 14. The system of claim 13, whereinthe device is enabled with a multi-tier authentication system in whichthe device and an application are configured with different levels ofprivacy and permissions.
 15. A multi-input system method for amultifunction mobile device comprising: accepting a gesture based inputto the mobile device, wherein the gesture comprises of a shape drawingaction performed in the air; presenting a split screen keyboard on themobile device, wherein the keyboard is positioned in multiple locationson the display of the keyboard, and wherein the keyboard displayedincludes character, numeric, and shape based input icons; accepting apassword using the split screen keyboard; presenting an authenticationconsists of a picture or shape tracing activity; requiring a match ofthe password to a pre-existing password entry; and requiring a match ofthe picture shape tracing activity to the outline of an image; andwherein at least one tracing activity comprises of the gesture basedinput.
 16. The method of claim 15, wherein the mobile device is enabledto accept a gesture based input and wherein said gesture based input isa shape drawn in the air in proximity to the mobile device.
 17. Themethod of claim 15, wherein the gesture comprises a shape tracingactivity received from a server.
 18. The method of claim 15, wherein thegesture based input provides a text based input used for anauthentication request or a text message.
 19. The method of claim 15,wherein the keyboard is movable to multiple areas on the mobile devicescreen and wherein the mobile device presents predictive text based onthe context of the mobile device including the location of the mobiledevice and the applications on the mobile device.
 20. The method ofclaim 15, wherein the device is enabled to accept a signature using atouch screen based handwriting input and an in the air signature gestureas recorded by the mobile device.